{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "# A Quasi-SVM in Keras\n",
    "\n",
    "**Author:** [fchollet](https://twitter.com/fchollet)<br>\n",
    "**Date created:** 2020/04/17<br>\n",
    "**Last modified:** 2020/04/17<br>\n",
    "**Description:** Demonstration of how to train a Keras model that approximates a SVM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Introduction\n",
    "\n",
    "This example demonstrates how to train a Keras model that approximates a Support Vector\n",
    " Machine (SVM).\n",
    "\n",
    "The key idea is to stack a `RandomFourierFeatures` layer with a linear layer.\n",
    "\n",
    "The `RandomFourierFeatures` layer can be used to \"kernelize\" linear models by applying\n",
    " a non-linear transformation to the input\n",
    "features and then training a linear model on top of the transformed features. Depending\n",
    "on the loss function of the linear model, the composition of this layer and the linear\n",
    "model results to models that are equivalent (up to approximation) to kernel SVMs (for\n",
    "hinge loss), kernel logistic regression (for logistic loss), kernel linear regression\n",
    " (for MSE loss), etc.\n",
    "\n",
    "In our case, we approximate SVM using a hinge loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "from tensorflow.keras.layers.experimental import RandomFourierFeatures\n",
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Build the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "\n",
    "model = keras.Sequential(\n",
    "    [\n",
    "        keras.Input(shape=(784,)),\n",
    "        RandomFourierFeatures(\n",
    "            output_dim=4096, scale=10.0, kernel_initializer=\"gaussian\"\n",
    "        ),\n",
    "        layers.Dense(units=10),\n",
    "    ]\n",
    ")\n",
    "model.compile(\n",
    "    optimizer=keras.optimizers.Adam(learning_rate=1e-3),\n",
    "    loss=keras.losses.hinge,\n",
    "    metrics=[keras.metrics.CategoricalAccuracy(name=\"acc\")],\n",
    ")\n",
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Prepare the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "\n",
    "# Load MNIST\n",
    "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
    "\n",
    "# Preprocess the data by flattening & scaling it\n",
    "x_train = x_train.reshape(-1, 784).astype(\"float32\") / 255\n",
    "x_test = x_test.reshape(-1, 784).astype(\"float32\") / 255\n",
    "\n",
    "# Categorical (one hot) encoding of the labels\n",
    "y_train = keras.utils.to_categorical(y_train)\n",
    "y_test = keras.utils.to_categorical(y_test)\n",
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "\n",
    "model.fit(x_train, y_train, epochs=20, batch_size=128, validation_split=0.2)\n",
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "I can't say that it works well or that it is indeed a good idea, but you can probably\n",
    " get decent results by tuning your hyperparameters.\n",
    "\n",
    "You can use this setup to add a \"SVM layer\" on top of a deep learning model, and train\n",
    " the whole model end-to-end."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "quasi_svm",
   "private_outputs": false,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}